How Does the Brain Represent Visual Scenes? A Neuromagnetic Scene Categorization Study

How are visual scenes represented in the brain during categorization? We acquired magnetoencephalography (MEG) data from nine healthy subjects who participated in a rapid natural scene categorization task. Scenes were presented in two different perspectives (aerial vs. terrestrial) and two different orientations (upright vs. inverted). We applied multivariate pattern classification to categorize scene categories from computational (spatial envelope (SpEn): [6]) and neural representations (MEG responses). Predictions of both types of classifiers (1) exceeded chance but performed worse than human subjects, and (2) were significantly correlated in their pattern of predictions, suggesting the relevance of low-level visual features during scene categorization. In general, the pattern of predictions and errors were not correlated with behavioral predictions. We also examined the influence of perspective and orientation on neural and computational representations by studying the generalization performance of classifiers across perspective and orientation. We compared within-perspective-and-orientation classifiers (trained and tested on the same perspective and orientation) with across-perspective (trained on one perspective and tested on another) and across-orientation classifiers (trained on one orientation and tested on another). We report several interesting effects on category-level and identity-level (dis)agreement between neural, computational, and behavioral ”views”. To our knowledge, this is the first study to examine natural scene perception across scene perspectives and orientations from neural, computational, and behavioral angles.